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Human  Performance 
Assessment  In  Training 
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Instructional  Method 


Learning  Domains 


z 


Cognitive  -  Intellectual  skills  (facts,  procedures, 
concepts,  rules,  principles) 


Psychomotor  -  complex  perceptual  -  motor  skills 

Affective  -  emotional  control,  stress  -  coping, 
attitudinal  predisposition  to  respond 


MOVES 


Learning  Stages  for 
Complex  Cognitive  Skills: 


1.  Novice  -  Rigid  adherence  to  rules;  poor  situational 
perception;  no  experience 


2.  Advanced  beginner  -  uses  guidelines  for  action,  still  limited 
situational  perception 

3.  Competent  -  better  view  of  big  picture;  plans  ahead;  but 
still  procedure  driven. 

4.  Proficient  -  holistic  view  and  able  to  extract  most  important 
elements  of  situation;  uses  maxims  (generalizable) 

5.  Expert  -  no  longer  relies  on  rules,  guidelines  or  maxims; 
intuitive  grasp  and  recognition  of  patterns  for  ease  of 
decision  action. 

Dryfus  &  Dryfus  (1980) 
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Learning  Stages  For 
Complex  Perceptual  Motor  Learning 


Cognitive  -  knowledge  based  (thinking,  verbal)  v 
understanding  of  task  and  strategies 

Associative  -  procedural  (rule-based)  understanding  of 
steps  or  task  sequence 

Autonomous  -(unconscious)  automatic  execution  of 
skilled  performance  with  high  accuracy  and  precision 


Fitts  and  Posner  (1967) 
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Skill  Acquisition  Theories 

Ackerman 


Patterns  of  correlation  coefficients  between  cognitive  knowledge 
and  perceptual  motor  performance  change  substantially  over  the 
course  of  training. 

Ackerman  (2007)  gives  the  following  reason  for  this  finding. 


Cognitive  aspects  of  the  task  are  more  influential  during  early  skill 
learning  -  so  perceptual-motor  components  have  greater 
influence  as  learning  progresses 
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First  Principles  of  Instruction 
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(Merrill  2007) 
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Instructional  Strategies  for  Complex 
Perceptual  Motor  Training 

•  Teaching  Strategies  for  Associative  Phase 

•  Provide  the  learner  an  opportunity  to  practice  the  procedure  (with  coaching  and 
feedback). 

•  Test  student  at  completion  of  practice  trials  at  appropriate  level  of  performance 
defined  in  the  learning  objectives. 

•  Complex  tasks  can  be  broken  down  into  key  components  for  deliberate  practice. 

•  Teaching  Strategies  for  the  Autonomous  Phase 

• 

•  1.  Continue  deliberate  practice  on  the  more  difficult  task  components. 

•  2.  Increase  the  speed  of  response  and  task  difficulty  progression.  (Challenge  the 
learner.) 

Selection  of  an  incorrect  strategy  can  impact  learning  effectiveness. 
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Instructional  Strategies 

These  strategies  are  based  upon  teaching  experience  and  are 
supported  by  skill  acquisition  studies: 


•  Provide  students  with  tutorial  information  regarding  the  task 
structure  and  salient  cues  to  pay  attention  to,  at  the  start  of 
instruction  -  but  reduce  or  eliminate  later. 

•  Demonstrate  correct  performance  to  student,  showing  correct 
response  sequence  and  timing. 

•  Reinforce  attention  to  salient  cues  during  practice  sessions,  and 
provide  feedback  of  performance  results. 

•  As  skill  becomes  more  automated,  provide  less  coaching  and 
verbal  support  during  task  performance 
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After  Action  Performance  Feedback  ; 


Fully  integrated  (with  a  training  system),  or  a  strap-on  automated 
performance  system,  that  records  performance,  reconstructs  and 
plays  back  training  sessions. 

•  Task-based  diagnostic  assessment  and  highlighting  of  key  mission 
events,  providing  immediate  post-mission  feedback  of  training 
results,  and  for  evaluation  of  training  effectiveness. 

—  Diagnostic  (corrective)  feedback  indexed  to  key  training  scenarios 
and  learning  objectives 

—  Cumulative  performance  data  for  tactics  development  and  for 
estimating  combat  readiness,  and  training  effectiveness 
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Evaluation  of  Training 


Kirkpatrick's  Model 

LEVEL  1-  Reaction  (attitude  -  opinion) 
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•  LEVEL  2  -  Learning  (achievement  test) 


•  LEVEL  3  -  Behavior  (job  performance) 

•  LEVEL  4  -  Results  (effectiveness  -  impact) 
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End  of  Course  evaluation 


z 


Used  to  determine  whether  or  not  student  has 
mastered  course  learning  objectives. 

Typically  measured  by  achievement  test,  based 
on  specified  knowledge  and  skills 


Objective  tests  -  usually  written  exams  can  be 
norm-based  or  criterion  based 
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«* 

Types  of  tests  : 

c 

H 

m 

•  Norm-based  -  most  common  form,  puts  all 
students  on  "normal  distribution"  using  class 
average,  or  some  other  form  of  comparative 
scale  for  distribution  scoring. 

•  Criterion-based  -  uses  percent  completion  or 
mastery  of  specific  learning  objectives  per 
student  against  preset 
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Placement 

Normal 


>f  a  specific  rating  on  : 
Distribution  Curve 

m 

Normal  Curve  Interpretation 


Rating  is  in  14th  percentile 
86%  of  ratings  are  above  this  one 
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Measurement  Issues 
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•  Internal  and  External  Experimental  validity 


-  Internal  validity:  did  treatment  (training)  make  the  difference 

-  External  validity:  will  this  treatment  work  in  other  similar  applications 
(generalizability  of  results) 

•  Measurement  Reliability  -  dependable,  error  free  measures 
(acceptable  performance  variability) 

•  Measurement  Validity  -  do  measures  make  sense  for 
intended  purpose.  Are  we  measuring  the  right  thing  and 
correctly  interpreting  measures  used. 
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Cognitive  Performance 

Classroom  exams  -  objective  and  essay 
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•  Academic  Laboratory  exams 


•  Simulation  and  Game  Exercises 


Problem  Solving  projects  (teamwork) 
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Perceptual-motor  performance  : 
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•  Reaction  times 

•  Response  rates 

•  Accuracy 

•  Precision  (performance  variation) 


Errors  and  deviations  from  a  criterion 
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Complex  perceptual  motor  i 

Common  measures  : 


•  Time  history 


•  Amplitude  distribution 


•  Frequency  domain 


•  Binary 


Time  on  target 

Accuracy  and  precision  transforms 

Time  out  of  tolerance 

Percent  time  in  tolerance 

Response  or  reaction  time 

Tracking  deviations  (overshoot  undershoot) 

Mean,  median,  mode 
Standard  deviation,  variance,  range 
Minimum/maximum  value 
Root-mean-squared  error 
Absolute  average  error 

Power  spectral  density  function 
Bandwidth,  peak  power,  low-high  frequency 
Wave  form  and  other  Transfer  functions 

Switch  activation  sequence  (right -wrong) 

Procedure  completion,  (omission,  commission 
errors) 


(Vreuls  and  Wooldridge,  1977) 
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METRICS  -  RMS 
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Way  of  calculating  the  root  mean  square 

The  RMS  of  a  collection  of  n  values 
X,™  =  s/W){xUX2,---,Xn} 


The  RMS  of  a  periodic  function  is  equal  to  the  RMS  of  one  period  of  the  function.  The  RMS  value  of  a  continuous 
function  or  signal  can  be  approximated  by  taking  the  RMS  of  a  series  of  equally  spaced  samples 


Basic  Skill  Acquisition  Progress  : 


Learning  Curve 


MOVES 


Basic  Skill  Acquisition  Progress  :q 
Measures  of  Precision 

:</> 


Marksmanship  Skill  Application 


The  US  Army  and  USMC  define  expert  marksmanship  as  hitting  36  or  more 
targets  out  of  40  on  a  pop-up  target  live  firing  range.  (FM  3-22.9  Rifle  Marksmanship,  2008; 

Marine  Corps,  2001) 

Basic  Marksmanship  Skills  --  The  Army  and  Marine  Corps  define  and  train  to 
five  basic  skill  components  of  rifle  marksmanship: 


1.  Trigger  pull,  2. breathing,  3. butt  stock  pressure,  4.weapon  cant,  and 
5. consistent  (stabile)  aiming  point. 

An  exploratory  study,  using  motion  capture  technology,  was  conducted  by 
our  graduate  students  US  Army  officers,  Majors  William  Platte  and  Johnny 
Powers.  The  purpose  was  to  measure  shooting  postural  performance  as  a 
fundamental  shooting  skill  --  in  order  to  objectively  determine  a  shooter's 
skill  level,  or  overall  expertise,  and  possibly  to  provide  a  diagnostic 
assessment  and  feedback  method. 
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Engagement  Skills  Trainer  (EST) 
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EST  After  Action  Review: 
Scoring  Accuracy:  Outcome  Metrics; 

m 
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Marksmanship  Skill 

Example  -  Correct  Prone  Position 


The  prone  position  provides  a  very 
steady  foundation  for  shooting 
and  presents  a  low  profile  for 
maximum  concealment.  However,  the 
prone  position  is  the  least  mobile  of 
the  shooting  positions  and  may 
restrict  a  Marine's  field  of  view  for 
observation.  In  this  position,  the 
Marine's  weight  is  evenly  distributed 
on  the  elbows,  providing  maximum 
support  and  good  stability  for  the  rifle. 


MCRP  3-01 A 
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Individual  Performance  Measures 
(Cognitive  -  Marksmanship) 

Explain  concept  of  firing  accuracy  (gun  sight 
calibration,  aiming  and  recoil  control) 

Show  instructor  correct  aiming  and  stock-weld 
procedure. 

Explain  how  correct  -  incorrect  breath  control  affects 
accuracy. 

Explain  the  effects  of  wind  and  distance  on  bullet 
trajectory 

Replay  -  critical  review  to  assess  the  quality  of 
tactical  decisions,  and  specific  decision  errors 


S3ACW 


z 

Motion  Capture:  Postural  Accuracy  : 

Skill  Process  Metric 
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Post  Processing 
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Conclusions  and  Recommendations 


We  are  attempting  to  define  and  validate  instructional  strategies  and  human  performance 
assessment  methods  for  complex  skill  training,  using  instrumented  marksmanship  training  as 
an  experimental  test  bed. 
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•  In  this  phase  of  our  study,  we  defined  instructional  strategies  that  distinguish  cognitive  from 
perceptual  motor  teaching  approaches,  with  the  aim  of  providing  improved  instruction  in  a 
variety  of  complex  perceptual  motor  tasks 

•  Based  on  human  performance  research  findings  from  the  literature  we  helped  clarify  the  use 
of  different  forms  of  feedback  that  are  applicable  to  various  stages  of  learning  complex  skills. 

•  We  sought  to  examine  new  forms  of  instrumentation  (like  motion  capture)  that  may  provide 
"process  oriented"  skill  metrics  for  skill  assessment  and  diagnostic  feedback. 

•  During  our  next  phases  of  research  we  will  be  addressing  issues  related  to  marksmanship 
training  in  more  immersive  environments  that  include  realistic  interactive  scenarios  (such  as 
engaging  moving  targets,  decision  making  and  combat  team  tactics) .  The  next  Phase  of 
research  also  will  identify  empirical  and  non  -  empirical  training  effectiveness  evaluation 
methods  and  analysis  templates. 
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